Skip to content

Horizon: let the template fork workers for supervisors inside the master - #7

Merged
tevkert merged 3 commits into
tevkert:mainfrom
aloware:pr/inline-supervisors
Nov 20, 2026
Merged

tevkert merged 3 commits into
tevkert:mainfrom
aloware:pr/inline-supervisors

Conversation

@volodymyr-aloware

Copy link
Copy Markdown
Contributor

Some apps run Horizon's supervisors inside the master process instead of one process each, to save the memory of a booted app per supervisor. Today such an app has to run queue-prefork stock: its own AddSupervisor binding fails the preflight, and nothing could fork its workers anyway, since workers are forked only by a unit's fork server at its supervisor loop's request.

This PR lets the template fork those workers directly, on the master's request. In our measurements the two savings add up: the in-process supervisors cost nothing extra, and every worker shares the template's compiled code.

What changes

WorkerSet (first commit, no behaviour change). The fork server's handling of horizon:work workers (fork, admit, pause state, cancel, signal, ready, reap) moves into src/Server/WorkerSet.php so that a second server can use it. UnitServer keeps its loop and passes it every worker message as before.

The template forks workers (second commit).

  • ForkedProcess sends its role in spawn. For role: worker, the template forks a horizon:work worker through its own WorkerSet, routes admit, pause_state, cancel and signal for worker ids, and reaps them. Anything else still forks a unit, so existing masters are unaffected.
  • The template TERMs the workers it forked when its master is gone (hosted: in masterExited(), before it exits or execs the stock master; spawned: when the master is gone or its channel closes), so none outlives it. The fork blocks the relayed signals until the child has reset its handlers.
  • Master::start() takes an inline flag, defaulting to the new config horizon_inline (QUEUE_PREFORK_HORIZON_INLINE). Inline mode skips Preflight's bindings check only, applies the off switches itself (as Wiring does before binding the package's command), and registers its own tick on MasterSupervisorLooped, since the app's command replaces PreforkHorizonCommand.
  • Master::workerProcess(SupervisorOptions) hands out a forked handle, or null for stock, with the unit breaker's rule kept per supervisor.
  • ForksWorkersFromTemplate is a trait for the app's ProcessPool subclass. createProcess() asks the master for a forked handle. Horizon restarts a stopped worker on the handle it has, so monitor() first replaces a stopped forked handle whose template is gone, or whose supervisor the breaker turned stock, with a new one.
  • queue-prefork:doctor checks this mode when horizon_inline is set.
  • README gets a "Supervisors inside the master" section (setup, fallbacks, and why horizon:purge must not run in this mode); CHANGELOG gets an Unreleased entry.

Limitations

  • In a master that runs most supervisors inline, a supervisor kept in its own process is exec'd as stock; AddSupervisor::createProcess() could use unitProcess() for it, but that is left out here.
  • When the breaker trips, workers still running keep their forked handle until they stop; they are replaced as they exit, so for a while such a pool mixes forked and stock workers.

Happy to rename the config key, split the PR, or move the trait wherever it fits better.

A unit's fork server forks, admits, signals and reaps the horizon:work
workers of its supervisor loop. That handling moves into WorkerSet,
unchanged, so another server can fork workers the same way. UnitServer
keeps the loop and passes it every worker message, as before.
Some apps run Horizon's supervisors inside the master process instead of
one process each. Their workers can now be forked as well: the master asks
its template for each worker (a spawn with role: worker), and the template
forks it through WorkerSet, with no fork server in between. The template
TERMs the workers it forked when its master is gone, so none outlives it.

On the master's side, Master::start() in inline mode
(queue-prefork.horizon_inline) skips the check for the app's own
AddSupervisor binding, applies the off switches itself and pumps on
MasterSupervisorLooped. Master::workerProcess() hands out forked handles
with a crash breaker per supervisor, and ForksWorkersFromTemplate lets the
app's process pool use them. Horizon restarts a worker on the handle it
has, so a stopped handle whose template is gone, or whose supervisor the
breaker turned stock, is replaced with a new one, forked again or stock.
The doctor reads horizon_inline as well.
@tevkert

tevkert commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Hi Volodymyr, thanks for the PR and the detailed write-up!

To be clear up front, I'm not against the change. It's careful work: it reuses the fork server's worker handling, setups that don't opt in are untouched, and the tests pass. Before merging, though, I'd like to understand what keeps you on inline supervisors, because once queue-prefork is on, most of what they save is already gone.

Rough numbers from the README benchmarks, for 10 supervisors and 40 workers (your app's sizes will differ, but the ratios should hold):

Setup Memory
Stock Horizon ~3.55 GB
Supervisors inside the master ~3.0 GB
Standard Horizon + queue-prefork ~0.73 GB
Both ~0.67 GB

A forked supervisor costs about 6 MB instead of ~56 MB, so on top of queue-prefork, running them inline saves about 6 MB each.

I went through something similar myself. Before queue-prefork, my Horizon ran with a trimmed boot that loaded only what Horizon and the workers needed. That gave sizable savings, but with queue-prefork they became a rounding error, so I went back to starting Horizon normally.

So is it memory for you, or something else: fewer processes, fewer Redis connections, something about how your supervisors run? If there's a reason to stay inline, I'm happy to take this, with a few small changes on top (mainly falling back to stock when a worker can't be spawned, and an integration test with real Horizon). If it's only memory, standard Horizon with queue-prefork gets you nearly all of it without maintaining your own Horizon subclasses.

@volodymyr-aloware

Copy link
Copy Markdown
Contributor Author

Thanks for looking at it so closely, and for the numbers.

Your per-supervisor figure holds for us: on top of queue-prefork, a supervisor run inline saves about 7 MB. What decides whether that matters is the ratio of supervisors to workers. At 10 supervisors and 40 workers it's a rounding error. But an idle forked worker costs about as much as a forked supervisor, so when the two counts are close, inline mode removes about half of what's left.

That's our case on dev machines, where RAM is what limits us: there we run roughly as many supervisors as workers, and inline mode takes queue-prefork's ~1.2 GB down to ~0.7 GB.

At around 100 of each, that's half the memory and half the Redis connections. Each supervisor process holds its own connections (Horizon's, for commands and its repository, plus its queue connection, which the autoscaler reads every loop), while inline supervisors share the master's. The process count drops by more than half, since with queue-prefork each supervisor is a supervisor process plus its fork server.

The inline subclasses predate queue-prefork in our app, so keeping them isn't new work for us.

Happy to add both changes: falling back to stock when a worker can't be spawned, and an integration test with real Horizon. I'll push them to this branch.

@tevkert

tevkert commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Thanks, that makes sense. With about one worker per supervisor the numbers look very different, and the Redis connections are a good point too. Go ahead and push both changes; I'll add a couple of small things on top, then merge and release it as 1.2.0.

A worker the template could not fork left its slot empty: Horizon
restarted it on the same handle, which asked the template again on every
restart. Its supervisor now turns stock (a UnitStock event), and the pool
execs a worker in its place.

The workbench runs Horizon's supervisors inside the master behind
HORIZON_INLINE, and HorizonInlineTest drives real Horizon that way: the
process tree, jobs, pause and continue, a killed worker, a failed fork
(a process limit on the template) and horizon:terminate. CI runs it as
its own Docker leg.
@volodymyr-aloware

Copy link
Copy Markdown
Contributor Author

Pushed both in b9dbcf3.

  • Stock fallback: when the template can't fork an inline worker, Master turns that supervisor stock (a UnitStock event, reason worker spawn failed: …), and ForksWorkersFromTemplate replaces the handle with an exec'd worker on the next loop. Before, the handle asked the template again on every restart and the slot stayed empty. A lost template stays on the existing path: every handle retires.
  • Integration test: the workbench runs its supervisors inside the master behind HORIZON_INLINE=true (four small classes in workbench/app/Horizon), and tests/Integration/HorizonInlineTest.php drives real Horizon that way: the process tree, jobs on forked workers, pause and continue, a killed worker, horizon:terminate, and a failed fork. The failure comes from prlimit --nproc=1: on the template alone, so the package has no test hook for it; without the fix, that test times out waiting for the stock worker. The class skips unless the stack runs inline. CI runs it as its own Docker leg, and docker/README.md has the commands.

Two things the test ran into, both handled in the workbench: Horizon holds back a worker restarted within the last minute, so the fallback test kills a worker no other test touches; and a paused master skips its supervisors' loops, so the workbench persists the inline supervisors on MasterSupervisorLooped while paused, or their records would expire after 30 seconds.

@tevkert

tevkert commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Thanks Volodymyr, both changes look good, and the integration test is great.

GitHub doesn't let me push to an organization's fork, so I opened #8 with your commits unchanged plus a few of mine; merging it marks this one merged too. The main additions: a shared crash breaker, an inline-only compatibility check for what the trait relies on, the package no longer replacing the app's horizon command in inline mode, and a comparison test against a stock inline stack. One note for your app: create the pool with the trait only once Master::start() returned true (the README says so now).

It also turned up a pause bug that affects every Horizon setup, fixed there too. I'll merge once CI is green and release 1.2.0.

tevkert added a commit that referenced this pull request Nov 20, 2026
Supervisors inside the Horizon master (#7)
@tevkert
tevkert merged commit b9dbcf3 into tevkert:main Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants